Performance metrics in an interactive computer simulation

ABSTRACT

A simulation mapping system and method for determining a plurality of performance metric values in relation to a training activity performed by a user in an interactive computer simulation, the interactive computer simulation simulating a virtual element comprising a plurality of dynamic subsystems. A processor module obtains dynamic data related to the virtual element being simulated in an interactive computer simulation station comprising a tangible instrument module. The dynamic data captures actions performed by the user on tangible instruments. The processor module constructs a dataset corresponding to the plurality of performance metric values from the dynamic data having a target time step by synchronizing dynamic data and by inferring, for at least one missing dynamic subsystems of the plurality of dynamic subsystems missing from the dynamic data, a new set of data into the dataset from dynamic data associated to one or more co-related dynamic subsystems.

TECHNICAL FIELD

The present invention relates to data communication and, moreparticularly, to data communication as required in the context oftraining through interactive computer simulations.

BACKGROUND

An interactive computer simulation system performs one or moreinteractive computer simulations. Each interactive computer simulationcomprises one or more virtual simulated elements each representing anactual system (e.g., multiple virtual aircraft systems each representingan actual aircraft). Each interactive computer simulation provides avirtual computer generated environment and various tangible instruments(or controls) in a simulation station to allow enactment of differentscenarios for the purpose of training one or more users (or trainees),using one or more of the virtual simulated elements, in the operationand/or understanding of the corresponding one or more actual systems.The virtual simulated element, or simulated element, is defined hereinas a simulated system. The simulated element is a virtual version thatsimulates, to the extent required by the interactive computersimulation, behavior of an actual system. The various tangibleinstruments accessible to the one or more users in the simulationstation replicate actual instruments or otherwise reproduce behavior ofthe actual instruments found in the actual system.

Different interactive computer simulation systems rely on differentstrategies for providing a training environment that suits giventraining needs. For instance, some interactive computer simulationstations embed real life instruments or systems parts (i.e., realavionics boxes or real chirurgical tool) while some others are madespecifically for the purpose of the interactive computer simulationsystem.

Making sure that the data required from the different subsystems, andthat need to be provided to the different subsystems, is properlycollected and exchanged is the challenge addressed by the presentinvention.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

In a first set of embodiments a first aspect is directed to a simulationmapping system for determining a plurality of performance metric valuesin relation to a training activity performed by a user in an interactivecomputer simulation, the interactive computer simulation simulating avirtual element comprising a plurality of dynamic subsystems. Thesimulation mapping system comprises a processor module that obtainsdynamic data related to the virtual element being simulated in aninteractive computer simulation station comprising a tangible instrumentmodule. The dynamic data captures actions performed by the user duringthe training activity on one or more tangible instruments of thetangible instrument module. The processor module also constructs adataset corresponding to the plurality of performance metric values fromthe dynamic data having a target time step by synchronizing dynamic datafrom at least two of the dynamic subsystems into the dataset consideringthe target time step, the at least two of the dynamic subsystems beingassociated to at least one common performance metric values from theplurality of performance metric values and by inferring, for at leastone missing dynamic subsystems of the plurality of dynamic subsystemsmissing from the dynamic data, a new set of data into the dataset fromdynamic data associated to one or more co-related dynamic subsystems,the co-related dynamic subsystems and the at least one missing dynamicsubsystems impacting at least one common performance metric values fromthe plurality of performance metric values.

The processor module may optionally obtain dynamic data from a pluralityof interactive computer simulation stations and constructs the datasethaving the target time step for the plurality of interactive computersimulation stations.

The processor module may optionally further provide the dataset as acommon standardized stream consumers, the consumers comprising a gradingsystem. The common standardized stream may comprise classificationinformation related to the plurality of performance metric values.

The processor module may optionally, when constructing the datasetcorresponding to the plurality of performance metric values from thedynamic data having the target time, add at least one simulated dynamicsubsystem missing from the dynamic data and an additional set of datainto the dataset from dynamic data associated to one or more co-relateddynamic subsystems, the co-related dynamic subsystems and the at leastone simulated dynamic subsystems impacting the at least one commonperformance metric values from the plurality of performance metricvalues.

The processor module may optionally apply a linear quadratic estimation(LQE) when constructing the dataset and/or a probabilistic directedacyclic graphical model when constructing the dataset.

In the first set of embodiments a second aspect is directed to a methodfor determining a plurality of performance metric values in relation toa training activity performed by a user in an interactive computersimulation, the interactive computer simulation simulating a virtualelement comprising a plurality of dynamic subsystems. The methodcomprises obtaining dynamic data related to the virtual element beingsimulated in an interactive computer simulation station comprising atangible instrument module. The dynamic data captures actions performedby the user during the training activity on one or more tangibleinstruments of the tangible instrument module. The method also comprisesconstructing a dataset corresponding to the plurality of performancemetric values from the dynamic data having a target time step bysynchronizing dynamic data from at least two of the dynamic subsystemsinto the dataset considering the target time step, the at least two ofthe dynamic subsystems being associated to at least one commonperformance metric values from the plurality of performance metricvalues and inferring, for at least one missing dynamic subsystem of theplurality of dynamic subsystems missing from the dynamic data, a new setof data into the dataset from dynamic data associated to one or moreco-related dynamic subsystems, the co-related dynamic subsystems and theat least one missing dynamic subsystems impacting the at least onecommon performance metric values from the plurality of performancemetric values.

The method may optionally further comprise obtaining dynamic data from aplurality of interactive computer simulation stations, whereinconstructing the dataset having the target time step is performed forthe plurality of interactive computer simulation stations.

The method may optionally further comprise providing the dataset as acommon standardized stream consumers, the consumers comprising a gradingsystem. The common standardized stream may optionally compriseclassification information related to the plurality of performancemetric values.

The method may optionally further comprise, when constructing thedataset corresponding to the plurality of performance metric values fromthe dynamic data having the target time, adding at least one simulateddynamic subsystem missing from the dynamic data and an additional set ofdata into the dataset from dynamic data associated to one or moreco-related dynamic subsystems, the co-related dynamic subsystems and theat least one simulated dynamic subsystems impacting the at least onecommon performance metric values from the plurality of performancemetric values.

Optionally, constructing the dataset may performed by applying a linearquadratic estimation (LQE) and/or by applying a probabilistic directedacyclic graphical model when constructing the dataset.

In a second set of embodiments a first aspect of is directed to aninteractive computer simulation system for training a user in aninteractive computer simulation in the performance of a task through atraining activity, the interactive computer simulation simulating avirtual element. The interactive computer simulation system comprises aninteractive computer simulation station and a processor module. Theinteractive computer simulation station comprises a tangible instrumentmodule, the user interacting with the tangible instrument module forcontrolling the virtual element in the interactive computer simulation.

The processor module obtains a plurality of performance metric datasetsrelated to the virtual element being simulated, the plurality ofperformance metric datasets representing results of the interactionsbetween the user and the tangible instrument module and, duringexecution of the interactive computer simulation, detects, in theplurality of performance metric datasets, a plurality of actualmaneuvers of the virtual element during the training activity,identifies one or more standard operating procedures (SOP) for thetraining activity from a plurality of the individually detected actualmaneuvers, provides, in real-time upon detection of the SOPs,information for display in the interactive computer simulation relatedthe SOPs.

The system may optionally further comprise a simulation mapping systemfor determining a plurality of performance metric values in relation tothe training activity performed by the user in the interactive computersimulation, the interactive computer simulation simulating the virtualelement comprising a plurality of dynamic subsystems. The plurality ofperformance metric datasets may be provided by the simulation mappingsystem.

The processor module may further obtain a scorecard related to thetraining activity to establish a list of the one or more SOPs ofinterest. The one or more SOPs may identify the plurality of theindividually detected actual maneuvers related thereto.

The plurality of performance metric datasets related to the virtualelement being simulated may be used to provide a grading scorecard forthe detected SOPs. The information for display in the interactivecomputer simulation related the SOPs may further comprise the gradingscorecard for the detected SOPs

The detected actual maneuvers may be logged for post-activitydebriefing.

In an optional embodiment, the processor module further obtains aplurality of expected maneuvers of the virtual element during thetraining activity, the plurality of expected maneuvers comprising aplurality of expected individual maneuvers expected and one or morenested maneuvers formed by more than one individual maneuvers from theplurality of expected individual maneuvers, computes the plurality ofperformance metric datasets to identify actual maneuvers of the virtualelement during the training activity, identifies and grades one or moreactual nested maneuvers against corresponding ones of the expectednested maneuvers and, notwithstanding performance of the actual nestedmaneuvers, identifies and grades a plurality of actual individualmaneuvers against the plurality of expected individual maneuvers.

In the second set of embodiments a second aspect is directed to aninteractive computer simulation station for training a user in aninteractive computer simulation in the performance of a task through atraining activity, the interactive computer simulation simulating avirtual element. The interactive computer simulation station comprises atangible instrument module, the user interacting with the tangibleinstrument module for controlling the virtual element in the interactivecomputer simulation and a processor module.

The processor module obtains a plurality of performance metric datasetsrelated to the virtual element being simulated, the plurality ofperformance metric datasets representing results of the interactionsbetween the user and the tangible instrument module and, duringexecution of the interactive computer simulation, detects, in theplurality of performance metric datasets, a plurality of actualmaneuvers of the virtual element during the training activity,identifies one or more standard operating procedures (SOP) for thetraining activity from a plurality of the individually detected actualmaneuvers and provides, in real-time upon detection of the SOPs,information for display in the interactive computer simulation relatedthe SOPs.

The interactive computer simulation station may further comprise anetwork interface nodule for receiving the plurality of performancemetric datasets from a simulation mapping system that determines aplurality of performance metric values in relation to the trainingactivity performed by the user in the interactive computer simulation.

The processor module may optionally further obtain a scorecard relatedto the training activity to establish a list of the one or more SOPs ofinterest. The one or more SOPs may further identify the plurality of theindividually detected actual maneuvers related thereto.

The plurality of performance metric datasets related to the virtualelement being simulated may be used to provide a grading scorecard forthe detected SOPs. The information for display in the interactivecomputer simulation related the SOPs may comprise the grading scorecardfor the detected SOPs.

In an optional embodiment, the processor module further obtains aplurality of expected maneuvers of the virtual element during thetraining activity, the plurality of expected maneuvers comprising aplurality of expected individual maneuvers expected and one or morenested maneuvers formed by more than one individual maneuvers from theplurality of expected individual maneuvers, computes the plurality ofperformance metric datasets to identify actual maneuvers of the virtualelement during the training activity, identifies and grades one or moreactual nested maneuvers against corresponding ones of the expectednested maneuvers and, notwithstanding performance of the actual nestedmaneuvers, identifies and grades a plurality of actual individualmaneuvers against the plurality of expected individual maneuvers.

In the second set of embodiments a third aspect is directed to a methodfor training a user in an interactive computer simulation in theperformance of a task through a training activity, the interactivecomputer simulation simulating a virtual element. The method comprisesin an interactive computer simulation station, providing a tangibleinstrument module to the user for controlling the virtual element in theinteractive computer simulation. The method also comprises obtaining aplurality of performance metric datasets related to the virtual elementbeing simulated, the plurality of performance metric datasetsrepresenting results of the interactions between the user and thetangible instrument module and, during execution of the interactivecomputer simulation at the interactive computer simulation station,detecting, in the plurality of performance metric datasets, one or moreactual maneuvers of the virtual element during the training activity,identifying one or more standard operating procedures (SOP) from thedetected actual maneuvers and displaying, in real-time upon detection ofthe SOPs, information in the interactive computer simulation related theSOPs.

The method may further optionally comprise determining, at a simulationmapping system, a plurality of performance metric values in relation tothe training activity performed by the user in the interactive computersimulation, the interactive computer simulation simulating the virtualelement comprising a plurality of dynamic subsystems. The plurality ofperformance metric datasets may be provided by the simulation mappingsystem.

The method may further optionally comprise obtaining a scorecard relatedto the training activity to establish a list of the one or more SOPs ofinterest. The one or more SOPs may further identify the plurality of theindividually detected actual maneuvers related thereto.

The plurality of performance metric datasets related to the virtualelement being simulated may be used to provide a grading scorecard forthe detected SOPs. The information for display in the interactivecomputer simulation related the SOPs may then optionally comprise thegrading scorecard for the detected SOPs

The method may further optionally comprise logging the detected actualmaneuvers and debriefing the training activity from the logged detectedactual maneuvers.

In some embodiments, the method may further optionally compriseobtaining a plurality of expected maneuvers of the virtual elementduring the training activity, the plurality of expected maneuverscomprising a plurality of expected individual maneuvers expected and oneor more nested maneuvers formed by more than one individual maneuversfrom the plurality of expected individual maneuvers, computing theplurality of performance metric datasets to identify actual maneuvers ofthe virtual element during the training activity, identifying and gradesone or more actual nested maneuvers against corresponding ones of theexpected nested maneuvers and, notwithstanding performance of the actualnested maneuvers, identifying and grading a plurality of actualindividual maneuvers against the plurality of expected individualmaneuvers.

In a third set of embodiments a first aspect is directed to aninteractive computer-based training system for assessing a trainingactivity performed by a user in an interactive computer simulation, theinteractive computer simulation simulating a virtual element. Thetraining system comprises an interactive computer simulation stationcomprising a tangible instrument module, the user interacting with thetangible instrument module for controlling the virtual element in theinteractive computer simulation and a processor module. The processormodule obtains a plurality of performance metric datasets related to thevirtual element being simulated the interactive computer simulationstation, the plurality of performance metric datasets representingresults of the interactions between the user and the tangible instrumentmodule, obtains a plurality of expected maneuvers of the virtual elementduring the training activity, the plurality of expected maneuvers,computes the plurality of performance metric datasets to identify actualmaneuvers of the virtual element during the training activity,identifies one or more failed actual maneuvers of the virtual elementduring the training activity against corresponding ones of the expectedmaneuvers and performs computational regression on the actual maneuversof the virtual element compared to the expected maneuvers of the virtualelement to identify one or more root causes of the failed actualmaneuvers, the computational regression being performed on the actualmaneuvers notwithstanding the corresponding expected maneuvers being metthereby.

The system may further optionally comprise a simulation mapping systemfor determining a plurality of performance metric values in relation tothe training activity performed by the user in the interactive computersimulation, the interactive computer simulation simulating the virtualelement comprising a plurality of dynamic subsystems, wherein theplurality of performance metric datasets is provided by the simulationmapping system.

The processor module may further optionally map, in real-time, each oneof the actual maneuvers of the virtual element during the trainingactivity on causal model for linking the one actual maneuver withprevious ones of the actual maneuvers. The processor module may thenoptionally associate a probability rating to the one or more root causesof the failed actual maneuvers considering the causal model.

The processor module may further optionally provide to an instructor ofthe user, in real-time, the one or more root causes of the failed actualmaneuvers.

The plurality of performance metric datasets related to the virtualelement being simulated may be used to provide a grading scorecard forthe actual maneuvers. The grading scorecard for the actual maneuvers maybe provided for display in the interactive computer simulation.

In the third set of embodiments a second aspect is directed to aninteractive computer simulation station for assessing a trainingactivity performed by a user in an interactive computer simulation, theinteractive computer simulation simulating a virtual element. Theinteractive computer simulation station comprises a tangible instrumentmodule, the user interacting with the tangible instrument module forcontrolling the virtual element in the interactive computer simulationand a processor module. The processor module obtains a plurality ofperformance metric datasets related to the virtual element beingsimulated the interactive computer simulation station, the plurality ofperformance metric datasets representing results of the interactionsbetween the user and the tangible instrument module, obtains a pluralityof expected maneuvers of the virtual element during the trainingactivity, the plurality of expected maneuvers, computes the plurality ofperformance metric datasets to identify actual maneuvers of the virtualelement during the training activity, identifies one or more failedactual maneuvers of the virtual element during the training activityagainst corresponding ones of the expected maneuvers and performscomputational regression on the actual maneuvers of the virtual elementcompared to the expected maneuvers of the virtual element to identifyone or more root causes of the failed actual maneuvers, thecomputational regression being performed on the actual maneuversnotwithstanding the corresponding expected maneuvers being met thereby.

The interactive computer simulation station may further comprise anetwork interface nodule for receiving the plurality of performancemetric datasets from a simulation mapping system that determines aplurality of performance metric values in relation to the trainingactivity performed by the user in the interactive computer simulation.

The processor module may further optionally map, in real-time, each oneof the actual maneuvers of the virtual element during the trainingactivity on causal model for linking the one actual maneuver withprevious ones of the actual maneuvers and associate a probability ratingto the one or more root causes of the failed actual maneuversconsidering the causal model.

The processor module may further optionally provide to an instructor ofthe user, in real-time, the one or more root causes of the failed actualmaneuvers.

The plurality of performance metric datasets related to the virtualelement being simulated may be used to provide a grading scorecard forthe actual maneuvers. The grading scorecard for the actual maneuvers maythen be provided for display in the interactive computer simulation.

In the third set of embodiments a third aspect is directed to a methodfor assessing a training activity performed by a user in an interactivecomputer simulation, the interactive computer simulation simulating avirtual element. The method comprises obtaining a plurality ofperformance metric datasets related to the virtual element beingsimulated the interactive computer simulation station, the plurality ofperformance metric datasets representing results of interactions of theuser with a tangible instrument module of an interactive computersimulation station, the user interacting with the tangible instrumentmodule for controlling the virtual element in the interactive computersimulation, obtaining a plurality of expected maneuvers of the virtualelement during the training activity, computing the plurality ofperformance metric datasets to identify actual maneuvers of the virtualelement during the training activity, identifying one or more failedactual maneuvers of the virtual element during the training activityagainst corresponding ones of the expected maneuvers and performingcomputational regression on the actual maneuvers of the virtual elementcompared to the expected maneuvers of the virtual element to identifyone or more root causes of the failed actual maneuvers, thecomputational regression being performed on the actual maneuversnotwithstanding the corresponding expected maneuvers being met thereby.

The method may optionally further comprise determining, at a simulationmapping system, a plurality of performance metric values in relation tothe training activity performed by the user in the interactive computersimulation, the interactive computer simulation simulating the virtualelement comprising a plurality of dynamic subsystems, wherein theplurality of performance metric datasets is provided by the simulationmapping system.

The method may optionally further comprise mapping, in real-time, eachone of the actual maneuvers of the virtual element during the trainingactivity on causal model for linking the one actual maneuver withprevious ones of the actual maneuvers. The method may then furthercomprise associating a probability rating to the one or more root causesof the failed actual maneuvers considering the causal model andproviding to an instructor of the user, in real-time, the one or moreroot causes of the failed actual maneuvers. The plurality of performancemetric datasets related to the virtual element being simulated may beused to provide a grading scorecard for the actual maneuvers. The methodmay optionally further comprise providing for display in the interactivecomputer simulation the grading scorecard for the actual maneuvers.

BRIEF DESCRIPTION OF THE DRAWINGS

Further features and exemplary advantages of the present invention willbecome apparent from the following detailed description, taken inconjunction with the appended drawings, in which:

FIG. 1 is a logical modular view of an exemplary interactive computersimulation system in accordance with the teachings of the presentinvention;

FIG. 2 is a flow chart of a first exemplary method in accordance withthe teachings of the present invention;

FIG. 3 is a flow chart of a second exemplary method in accordance withthe teachings of the present invention;

FIG. 4 is a flow chart of a third exemplary method in accordance withthe teachings of the present invention;

FIG. 5 is a flow and nodal operational chart 5000 in accordance with theteachings of the present invention;

FIG. 6 is a logical modular view of an exemplary interactive computersimulation system in accordance with the teachings of the presentinvention;

FIG. 7 is a logical representation of a first training activity inaccordance with the teachings of the present invention; and

FIG. 8 is a logical representation of a second training activity inaccordance with the teachings of the present invention.

DETAILED DESCRIPTION

Reference is now made to the drawings in which FIG. 1 shows a logicalmodular view of an exemplary interactive computer simulation system 1000in accordance with the teachings of the present invention. Theinteractive computer simulation system 1000 performs one or moreinteractive computer simulations. Each interactive computer simulationcomprises one or more virtual simulated elements each representing anactual system (e.g., multiple virtual aircraft systems each representingan actual aircraft). Each interactive computer simulation provides avirtual environment and various tangible instruments (or controls) toallow enactment of different scenarios for the purpose of training oneor more users (or trainees), using one or more of the virtual simulatedelements, in the operation and/or understanding of the corresponding oneor more actual systems. The virtual simulated element, or simulatedelement, is defined herein as a simulated system, and may furthercomprise multiple simulated dynamic subsystems, or dynamic subsystems.The simulated element is a virtual version that simulates, to the extentrequired by the interactive computer simulation, behavior of an actualsystem. Correspondingly, each of the simulated dynamic subsystems of thesimulated element is a virtual version, to the extent required but theinteractive computer simulation, behavior of actual subsystems of theactual system.

In the depicted embodiment of FIG. 1, the interactive computersimulation system 1000 comprises an interactive computer simulationstation 1100 for controlling at least one of the virtual simulatedelements from the computer simulation executed on the interactivecomputer simulation system 1000. The interactive computer simulationsystem 1000 typically comprises multiple simulation stations (not shown)that each allow one or more users to interact to control a virtualsimulated element in one of the interactive computer simulation(s) ofthe interactive computer simulation system 1000. The interactivecomputer simulation system 1000 also comprises a debriefing station 1600and a monitoring station 1700 also sometimes referred to as anInstructor Operating Station (IOS). The monitoring stations may beprovided for allowing various management tasks (not shown) to beperformed in the interactive computer simulation system 1000. The tasksassociated with the monitoring station 1700 allow for control and/ormonitoring of one or more ongoing interactive computer simulations. Forinstance, the monitoring station 1700 may be used for allowing aninstructor to participate to the interactive computer simulation andpossibly additional interactive computer simulation(s). In someembodiments, the monitoring station 1700 is provided with theinteractive computer simulation station 1100 (1700A). In otherembodiments, the monitoring station 1700 may be co-located with theinteractive computer simulation station 1100, e.g., within the same room(1700B) or simulation enclosure or remote therefrom (1700C), e.g., indifferent rooms or in different locations connected through a network1400. Skilled persons will understand the many instances of themonitoring station 1700 may be concurrently provided in the interactivecomputer simulation system 1000. The monitoring station 1700 may providea computer simulation management interface, which may be displayed on adedicated monitoring station 1700 user interface 1750 or the GUI module1150. The monitoring station 1700, in some embodiments, is provided asthe GUI 1750 on a portable computing device (e.g., smartphone, tablet,portable computer or the like).

When multiple simulation stations 1100 are present in the system 1000,the monitoring station 1700 may present different views of the computerprogram management interface (e.g., to manage different aspectstherewith) or they may all present the same view thereof. The computerprogram management interface may be permanently shown on a first of thescreens of the monitoring station 1700 display module while a second ofthe screens of the monitoring station 1700 display module shows a viewof the interactive computer simulation (i.e., adapted view consideringcharacteristics of the second screen). The computer program managementinterface may also be triggered on the monitoring station 1700, e.g., bya touch gesture and/or an event in the interactive computer program(e.g., milestone reached, unexpected action from the user, or actionoutside of expected parameters, success or failure of a certain mission,etc.). The computer program management interface may provide access tosettings of the interactive computer simulation and/or of the simulationstation 1100. A virtualized monitoring station may also be provided tothe user (e.g., through the GUI module 1150) on a main screen, on asecondary screen or a dedicated screen.

In some embodiments, the interactive computer simulation system 1000comprises a debriefing station 1600. The debriefing station 1600 issometimes referred to as a Brief and Debrief System (BDS). Thedebriefing station 1600 may provide functionalities also provided by themonitoring station 1700 in the context of debriefing past sessionsthereat. For instance, when monitoring station 1700 and/or debriefingstation 1600 functionalities are provided through the interactivecomputer simulation station 1100, the GUI module 1150/1650/1750 mayfurther be used to monitor and control one or more ongoing or recordedinteractive computer simulation (e.g., triggering/monitoring eventsand/or selecting a perspective from which to view the ongoing orrecorded chain of events of one or more interactive computersimulation).

The simulation station 1100, the monitoring station 1700 and thedebriefing station 1600 may be connected via a network 1400, via directconnections or a mix of direct and network connections. In the depictedexample of FIG. 1, the simulation station 1100 is a distinct simulationstation while, in some embodiments, the simulation station 1100 may beintegrated with one or more of the simulation stations. Various networklinks may be implicitly or explicitly used in the context of the presentinvention. While a link may be depicted as a wireless link, it couldalso be embodied as a wired link using a coaxial cable, an opticalfiber, a category 5 cable, and the like. A wired or wireless accesspoint (not shown) may be present on links. Likewise, any number ofrouters and/or switches (not shown) may be present on links, which mayfurther transit through the Internet.

In the depicted example of FIG. 1, the simulation station 1100 comprisesa memory module 1120, a processor module 1130 and a network interfacemodule 1140. The processor module 1130 may represent a single processorwith one or more processor cores or an array of processors, eachcomprising one or more processor cores. In some embodiments, theprocessor module 1130 may also comprise a dedicated graphics processingunit 1132. The dedicated graphics processing unit 1132 may be required,for instance, when the interactive computer simulation system 1000performs an immersive simulation (e.g., pilot training-certified flightsimulator), which requires extensive image generation capabilities(i.e., quality and throughput) to maintain expected realism of suchimmersive simulation. Typically, each of the monitoring station 1700and/or debriefing station 1600 comprise a memory module similar to 1120,a processor module similar to 1130 having a dedicated graphicsprocessing unit similar to 1132, a network interface similar to 1140 anda bus similar to 1170, which have not been replicated on FIG. 1 for thesake of readability. In some embodiments, the monitoring station 1700and/or debriefing station 1600 may also comprise an instrument modulesimilar to 1160 and a simulation mapping system similar to 1800A.

The memory module 1120 may comprise various types of memory (differentstandardized or kinds of Random Access Memory (RAM) modules, memorycards, Read-Only Memory (ROM) modules, programmable ROM, etc.). Thenetwork interface module 1140 represents at least one physical interfacethat can be used to communicate with other network nodes. The networkinterface module 1140 may be made visible to the other modules of thesimulation station 1100 through one or more logical interfaces. Theactual stacks of protocols used by the physical network interface(s)and/or logical network interface(s) 1142, 1144, 1146, 1148 of thenetwork interface module 1140 do not affect the teachings of the presentinvention. The variants of processor module 1130, memory module 1120 andnetwork interface module 1140 usable in the context of the presentinvention will be readily apparent to persons skilled in the art.

A bus 1170 is depicted as an example of means for exchanging databetween the different modules of the simulation station 1100. Thepresent invention is not affected by the way the different modulesexchange information between them. For instance, the memory module 1120and the processor module 1130 could be connected by a parallel bus 1170,but could also be connected by a serial connection or involve anintermediate module (not shown) without affecting the teachings of thepresent invention.

Likewise, even though explicit mentions of the memory module 1120 and/orthe processor module 1130 are not made throughout the description of thevarious embodiments, persons skilled in the art will readily recognizethat such modules are used in conjunction with other modules of thesimulation station 1100 to perform routine as well as innovative stepsrelated to the present invention.

In the depicted example of FIG. 1, the interactive computer simulationsystem 1000 comprises a simulation mapping system 1800A, 1800B. Asfurther explained and exemplified below, the simulation mapping system1800A, 1800B gathers, processes, converts and/or sends dynamic datarelated to the interactive computer simulation, typically in the form ofone or more streams of information, from the dynamic system and dynamicsubsystems of the interactive computer simulation system 1000 (e.g.,instrument module 1160, user interface 1150/1650/1750, video recordersystem, simulation engine, simulation station's security system, etc.).The dynamic data is typically generated during the interactive computersimulation in relation to the simulated element along a sessiontimeline.

In some embodiments, the simulation mapping system 1800A, 1800Bcomprises a local simulation mapping system 1800A, in each of theinteractive computer simulation stations 1100, and a coordinatingsimulation mapping system 1800B for the interactive computer simulationsystem 1000. In some embodiments, the coordination aspects of thesimulation mapping system 1800A, 1800B are distributed between thedifferent local simulation mapping systems 1800A. In some embodiments,the local aspects of the simulation mapping system 1800A, 1800B areperformed from the coordinating simulation mapping system 1800B (e.g.,through access to a storage system 1500 and/or to the different elementsof the interactive computer simulation system 1000).

In some embodiments, the simulation mapping system 1800B comprise amemory module similar to 1120, a network interface similar to 1140 and abus similar to 1170, which have not been replicated on FIG. 1 for thesake of readability. The simulation mapping system 1800A, 1800B may relyon the processor module 1130 to process and/or convert the dynamic data.The simulation mapping system 1800A, 1800B may also comprise, inaddition or alternatively, a processor module 1830 to process and/orconvert the dynamic data. The processor module 1830 may further comprisea dedicated graphics processing unit similar to 1132. The processormodule 1830 may also comprise a dedicated real-time processing unit (notshown) to process and/or convert at least some of the dynamic data. Thededicated real-time processing unit provides enhanced capabilities tosupport real-time processing or real-time processing priority. Thededicated real-time processing unit may be required, for instance, whenthe interactive computer simulation system 1000 performs an immersivesimulation (e.g., pilot training-certified flight simulator), which mayrequire the dynamic data to be timely processed and/or converted tomaintain expected realism of such immersive simulation. In someembodiments, the processor module 1830 is partly or completelyintegrated in a cloud-based processing service. The processor module1130, when used in the context of the simulation mapping system 1800,may therefore comprise capabilities to interact and/or manage thecloud-based processing service through the network interface 1140.

The simulation mapping system 1800A, 1800B may further comprise (notshown) an environment tracking module, which may be used to capture oneor more feed of images and/or environmental data from the interactivecomputer simulation station 1100. For instance, the environment trackingmodule may comprise one or more 360-degree camera and/or a plurality ofcameras throughout the interactive computer simulation station 1100 toprovide a choice of perspectives therein. For instance, the perspectivesoffered through the cameras may be set to cover as many criticallocations in the interactive computer simulation station 1100 (e.g.,position of the hands of trainee(s), readings or settings on one or moreof the instruments of the instrument module 1160 and/or determination ofa position of one or more instruments, tracking of the trainee(s)' gazeor other body parts, etc. The environment tracking module may alsocomprise one or more sound recorders (e.g., for conversations in thesimulation station as well as with outside elements), one or morethermometer, one or more biometric readers (e.g., trainee(s)' statusreadings, gaze detector, sleepiness detector, etc.), smoke or othervisual impairment detector, etc.)

The interactive computer simulation system 1000 comprises a storagesystem 1500 for, among other aspects, collecting dynamic data inrelation to the dynamic system and dynamic subsystems while theinteractive computer simulation is performed. The dynamic data stored inthe storage system 1500 comprises dynamic data necessary for thesimulation mapping system 1800A, 1800B as well as results from theprocessing and/or converting performed by the simulation mapping system1800A, 1800B. FIG. 1 shows examples of the storage system 1500 as adistinct database system 1500A, a distinct module 1500B of the computersystem 1110, a sub-module 1500C of the memory module 1120 of thesimulation station 1100 and/or a storage system 1500D comprises in thesimulation mapping system 1800A, 1800B. The storage system 1500 may alsocomprise storage modules (not shown) on the monitoring station 1700and/or debriefing station 1600. The storage system 1500 may bedistributed over different systems A, B, C, D and/or the monitoringstation 1700 and/or debriefing station 1600 or may be in a singlesystem. The storage system 1500 may comprise one or more logical orphysical as well as local or remote hard disk drive (HDD) (or an arraythereof). The storage system 1500 may further comprise a local or remotedatabase made accessible to the simulation station 1100 by astandardized or proprietary interface or via the network interfacemodule 1140 (e.g., cloud-based storage service). In some embodiments,the storage system 1500 stores the dynamic data and/or theprocessed/converted data in relation to the simulation mapping system1800A, 1800B in a cloud-based storage service. The variants of storagesystem 1500 usable in the context of the present invention will bereadily apparent to persons skilled in the art.

The interactive computer simulation station 1100 may comprise agraphical user interface (GUI) module 1150 that may be used to visualizevirtual dynamic subsystems from the virtual simulated element. The GUImodule 1150 may comprise one or more display screens such as a wired orwireless flat screen, a wired or wireless touch-sensitive display, atablet computer, a portable computer or a smart phone.

Users of the interactive computer simulation system 1000 (e.g., users ofthe simulation stations 1100) interact in the interactive computersimulation to control a virtual simulated element in a computergenerated environment of the interactive computer simulation system 1000(e.g., instructors or experts, trainees such as a pilot and co-pilot, adriver, an operator, a surgeon, a flight investigator, a traininganalyst, a flight analyst, etc.). Examples of virtual simulated elementsinclude a simulated aircraft system, a simulated ground vehicle system,a simulated spacecraft or space station system, a simulated control roomsystem, unmanned vehicle or drone, simulated human mannequin, etc.Examples of virtual dynamic subsystems vary depending on the virtualsimulated element. In the example of a simulated aircraft system,typical virtual dynamic subsystems may include virtual hydraulicsystems, virtual communication systems, virtual display systems, virtualwiring systems, virtual in-flight entertainment systems, virtual fuelsystems, virtual lighting systems, virtual rudder system, virtual flapsystem, virtual landing gear system, etc. In the example of a simulatedliving system, typical virtual dynamic subsystems may include bloodsystem, digestive system immunity response system, lymphatic system,nervous system, biometric data such as temperature, blood pressure andother related physical data, etc. When a trainee or user is involved,actual measurements of biometric data may also be recorded (e.g., forsubsequent correlation with other recorded data). For instance,biometric data from a pilot interacting in a computer simulation withone or more tangible instruments at the simulation station 1100 may berecorded (such as temperature, blood pressure and other related physicaldata). As a skilled person would appreciate, most virtual subsystems aredirectly or indirectly affected by interactions of the user with one ormore tangible instruments that allow the user to interact (e.g., providedifferent commands in order to control the virtual simulated element)during the interactive computer system in the computer generatedenvironment. Some other virtual subsystems may be affected by timeelapsed during the interactive computer system and may further take intoaccount the interactions of the user with one or more tangibleinstruments. For instance, in the example of a simulated aircraftsystem, a virtual aircraft structure subsystem may comprise one or morevirtual mechanical components. Failure of any one of virtual mechanicalcomponents, or the virtual aircraft structure subsystem altogether, maybe based on accumulated mechanical stress considering use time (e.g.,number of flights and operating hours) and also based on maneuverscaused by the pilot manipulating the one or more tangible instruments.

The tangible instrument provided by the instrument modules 1160 aretightly related to the element being simulated. In the example of thesimulated aircraft system, typical instruments include various switches,levers, pedals and the like accessible to the user for controlling theaircraft in the interactive computer simulation. Depending on the typeof simulation (e.g., level of immersivity), the tangible instruments maybe more or less realistic compared to those that would be available inan actual aircraft. For instance, the tangible instrument provided bythe module 1160 may replicate an actual aircraft cockpit where actualinstruments found in the actual aircraft or physical interfaces havingsimilar physical characteristics are provided to the user (or trainee).As previously describer, the actions that the user or trainee takes withone or more of the tangible instruments provided via the instrumentmodule 1160 (modifying lever positions, activating/deactivatingswitches, etc.) allow the user or trainee to control the virtualsimulated element in the interactive computer simulation. In the contextof an immersive simulation being performed in the interactive computersimulation system 1000, the instrument module 1160 would typicallysupport a replicate of an actual instrument panel found in the actualsystem being the subject of the immersive simulation. In such animmersive simulation, the dedicated graphics processing unit 1132 wouldalso typically be required. While the present invention is applicable toimmersive simulations (e.g., flight simulators certified for commercialpilot training and/or military pilot training), skilled persons willreadily recognize and be able to apply its teachings to other types ofinteractive computer simulations.

In some embodiment, an optional external input/output (I/O) module 1162and/or an optional internal input/output (I/O) module 1164 may beprovided with the instrument module 1160. Skilled people will understandthat any of the instrument modules 1160, 1260 and/or 1360 may beprovided with one or both of the I/O modules such as the ones depictedfor the computer system 1000. The external input/output (I/O) module1162 of the instrument module 1160, 1260 and/or 1360 may connect one ormore external tangible instruments (not shown) therethrough. Theexternal I/O module 1162 may be required, for instance, for interfacingthe interactive computer simulation system 1000 with one or moretangible instrument identical to an Original Equipment Manufacturer(OEM) part that cannot be integrated into the computer system 1100(e.g., a tangible instrument exactly as the one that would be found inthe actual system subject of the interactive simulation). The internalinput/output (I/O) module 1164 of the instrument module 1160 may connectone or more tangible instruments integrated with the instrument module1160. The I/O 1162 may comprise necessary interface(s) to exchange data,set data or get data from such integrated tangible instruments. Theinternal I/O module 1164 may be required, for instance, for interfacingthe interactive computer simulation system 1100 with one or moreintegrated tangible instrument identical to an Original EquipmentManufacturer (OEM) part (e.g., a tangible instrument exactly as the onethat would be found in the actual system subject of the interactivesimulation). The I/O 1164 may comprise necessary interface(s) toexchange data, set data or get data from such integrated tangibleinstruments.

In some embodiments, a simulation plan may further be loaded (not shown)from the storage system 1500 in relation the interaction computersimulation that involves the virtual simulated element. The simulationplan may comprise a training plan, a lesson plan or a scenario-basedplan (e.g., with specific or dynamic objectives to be reached). Thesimulation plan may also be used alternatively or additionally to setthe period of time covering simulated events from the interactivecomputer simulation related to the selected virtual subsystem.

The interactive computer simulation system 1000 is typically used totrain personnel for complex and/or risky operations. Each interactivecomputer simulation provides a virtual environment and various tangibleinstruments (or controls) to allow enactment of different scenarios forthe purpose of training one or more users (or trainees), using one ormore of the virtual simulated elements, in the operation and/orunderstanding of the corresponding one or more actual systems. In somesituations, real-life training is simply not possible because the targetscenario cannot be enacted safely in the real-life (e.g., militarymission, rescue mission, medical treatment or operation, etc.). In othersituations, it is impractical and/or too costly to enact the trainingscenario in real-life. The interactive computer simulation system 1000alleviates the risks and allows for repeated training. The interactivecomputer simulation system 1000 also limits the overall costs oftraining when compared to real-life training. Evaluating the performanceof the trainee, while it is sometimes only useful, may be criticallyimportant (e.g., evaluating preparedness before a mission, certifyingcompetences for license purposes, etc.).

Typically, an evaluation of a trainee in the context of the interactivecomputer simulation system 1000 consists of an assessment by aninstructor based on an interpretation of collected information (e.g.,stored dynamic data associated with different events) as well as onvisual subjective observations performed by the instructor during thesimulation. While it is agreed that a certain level of subjectivity isinherent to most if not all evaluations, there is a perceived risk thatthe competences of the trainees may not be properly and systematicallyassessed. For instance, two different instructors may make differentvisual observations and interpret the same collected informationdifferently. Similarly, quality and/or completeness of the collecteddata may not sufficient to properly assess performance.

For instance, different interactive computer simulation stations 1100may comprise slightly different versions of the tangible instrumentmodule 1160 for a single virtual element, leading to differences in thecollected dynamic data (e.g., different avionics components in tworeplicated cockpit of different aircraft simulators for the same realaircraft). Furthermore, some data from dynamic subsystems necessary forthe purpose of evaluation of may not be collected at all (e.g., becausethe data is not exposed to the rest of the virtual environment). Forinstances, data collected for an aircraft yoke may comprise simulatedhydraulic pressure levels in different affected subsystems withoutcomprising the angular position of the actual hardware yoke in thereplicated cockpit, or vice-versa. Synchronization of the dynamic datamay also create discrepancies in the quality of the resultingevaluation. Typically, the simulation of each of the systems andsubsystems requires clock signals (or time steps) for the purpose ofsynchronization. While a subsystem typically keeps a single time stepthroughout a given interactive computer simulation, the time step fordifferent subsystems may be different. Conversely, for data collectionefficiency purposes, the time at which the dynamic data is collected isnot typically constant throughout the simulation and may not either bethe same as the initially defined time steps. This discrepancy leads todifferent integration steps, which, for dynamic systems and subsystems,may induce states divergence. For non-dynamic systems, the difference intime steps induces time delays that might have critical impact on thestates being recreated. As another example, some simulators will fullyvirtualize the tangible instrument 1160 output inside the interactivesimulation station 1100 and collect the dynamic data in the form ofdigital value accordingly (i.e., store a vector of flight-relatedinstructions instead of storing a vector of angular positions for theaircraft yoke). The virtualization made of any given instrument isdependent on technologies available at the time of the development ofthe virtual element and, when a real instrument is integrated in areplicated environment, is also dependent on the technologies used bythe manufacturer of the instrument. The resulting collected data maytherefore present disparities considering the manner in which thevirtualization has been made.

Reference is now made to FIGS. 5 to 8. FIG. 5 depicts an example of dataexchange in the context of training of one or more users in completionof one or more training activities. FIG. 6 provides an exemplary modularview of the system 1000 where multiple interactive training simulationstations (1200 and 1300) are depicted. Examples of training activitiesare depicted respectively on FIG. 7 (7000) and FIG. 8 (8000), inrelation to flight simulations. Of course, skilled persons willacknowledge that the teachings presented herein are applicable to manydifferent types of interactive simulations and training activities(e.g., flight, land and/or marine vehicle, healthcare-related element,etc.). On FIG. 7, the virtual element is a simulated aircraft 7010 on aflight path 7020. The training activity 7000 consists for one or moretrainees to land the virtual aircraft 7010. In the interactive computersimulation station 1100, the trainees interact with the tangibleinstrument module 1160 to control the virtual aircraft 7010. Differentstages 7100, 7200, 7300, 7400, 7500 are defined for the trainingactivity and different standard operation procedures (SOPs) 7110, 7120,7210, 7220, 7310, 7320, 7410, 7510 for the stages.

On FIG. 8, the training activity 8000 consists for one or more traineesto follow a defined flight trajectory during which the virtual aircraft7010 is expected to perform against a predefined pattern. In theinteractive computer simulation station 1100, the trainees interact withthe tangible instrument module 1160 to control the virtual aircraft7010. Different manoeuvers are measured at different points 1, 2, 3, 4,5, 6, 7, 8 and 9 for the training activity. On the depicted example8000, first grades are awarded for each of the maneuvers (e.g., between1 and 2, between 2 and 3, between 3 and 4, etc.) and a grade is awardedfor the complete training activity 8000, which may also be referred toas a nested maneuver (i.e., a maneuver that consists of multipleindividual maneuvers performed in a specific manner and/or order).

In FIG. 5, a simulator data acquisition 5100 represents the collectionof “raw” dynamic data related to one or more interactive computersimulations. For instances, it may represent flight telemetries relatedto an aircraft in a flight simulation. The simulator data acquisition5100 of the dynamic data happens at or from the interactive computersimulation station 1100. For instance, dynamic data related to dynamicsystems and dynamic subsystems is collected, without analysis orsynchronization (e.g., as it is produced or emitted from differentcomponents of the interactive computer simulation). Thesimulation-related data is sent 5710 to a data frame processor 5200.While a single arrow is shown 5710, it should be understood that thesimulation-related data is continually provided to the data frameprocessor 5200 as it is collected. The data frame processor 5200 builds5810 simulation data frames from the received data. For instance, in theexample of a flight simulation, building 5810 the simulation data framesat the data frame processor 5200 may involve converting raw flighttelemetries into a synchronized time series required for eventdetection. An example of how building 5810 the simulation data framescan be performed is provided with particular reference to FIG. 2hereinbelow. Again, it should be understood that the simulation dataframes are continually provided from the data frame processor 5200,e.g., in accordance with a target time step. The simulation data framescomprise performance metric values related to the ongoing interactivecomputer simulation.

Different consumers may be interested in the simulation data frames. Theexample of FIG. 5 shows a simulation event detection 5300 entity and atraining event detection 5400 entity, which are involves in assessmentof performance of one or more users during the interactive computersimulation. Examples of consumers, amongst others, may comprise amaintenance agent for the interactive computer simulation station (e.g.,in relation to QTGs) and accounting systems (e.g., in relation tooccupancy and/or costs of operation). In the example of FIG. 5, thesimulation data frames are sent 5720, 5722 respectively towards asimulation event detection 5300 entity and a training event detection5400 entity. Note that it may be pushed through a broadcast and/ormulticast mechanism and/or could be pulled by the consumers.

The simulation event detection 5300 entity is then shown detecting 5820a simulation event in the received data. The detection 5820 may be theresult of processing a single of the received frames or the result ofprocessing multiple frames, whether received consecutively or not. Forinstance, the detected event may be related to a general parameter ofthe virtual element being simulated (e.g., speed, altitude, temperature,ambient conditions in the interactive computer simulation station,etc.). The detected event may be the result of processing the data frameto detect flight event (ex: a flight exceedance or any other flag onflight status). It may be necessary to persistently collect the detectedevent 5910 (e.g., depending on the nature of the event or the relationof the event to the training activity). The related data is then sent5730 for storage at an analytic storage 5400 (e.g., part of the storagesystem 1500). The analytic storage 5400 may be a database (e.g., SQLcompatible) and may further be used to hold scorecards and otheranalytic data related to training activities.

The training event detection 5400 entity is then shown detecting 5830 atraining event in the received data. The detection 5830 may be theresult of processing a single of the received frames or the result ofprocessing multiple frames, whether received consecutively or not. Forinstance, the detected event may be related to a standard operationprocedure (SOP) as depicted on FIG. 7, an individual maneuver from FIG.8 or the nested maneuver 8000 altogether. It may be necessary topersistently grade the detected event 5930 (e.g., depending on thenature of the event or the relation of the event to the trainingactivity). The related data is then sent 5740 for analysis by a gradingcalculator 5600 that may further require additional simulation relatedevents to properly compute 5840 the grade and build a related scorecard.

Additional simulation related data may then be requested 5750 from theanalytic storage 5400 and returned 5760 thereby, if anything relevantexists (e.g., during computational regression following a causal modelto identify a root cause, to grade a nested maneuver, etc.). The relatedscorecard is then sent 5770 for storage at the analytic storage 5400.

The simulation event detection 5300 entity may also more selectivelydetect 5920 a simulation event in the received data consideringscorecards related to the training activity (e.g., a specific period ortrigger point from the interactive computer simulation). The simulationevent detection 5300 may therefore requests scorecards 5880 from theanalytic storage 5400, which returns 5822 relevant ones, if anytherefrom. The detection 5822 may be the result of processing a singleof the received frames or the result of processing multiple frames,whether received consecutively or not. For instance, the detected eventmay be related to a general parameter of the virtual element beingsimulated in relation to the training activity (e.g., speed, altitude,temperature whereas ambient conditions in the interactive computersimulation station may not be relevant, etc.). The detected event may bethe result of processing the data frame to detect flight event (ex: aflight exceedance or any other flag on flight status). It may benecessary to persistently collect the detected event 5920 (e.g.,depending on the nature of the event or the relation of the event to thetraining activity). The related data is then sent 5790 for storage atthe analytic storage 5400. In some embodiments, the simulation mappingsystem 1800A, 1800B comprises the data frame processor 5200, thesimulation event detection 5300, the analytic storage 5400, the trainingevent detection 5500 and the grading calculator 5600.

Reference is now made to FIGS. 1, 2 and 5 to 8 depicting a first set ofembodiments. In the depicted example, the simulation mapping system 1800is for determining a plurality of performance metric values in relationto a training activity performed by a user in an interactive computersimulation. The interactive computer simulation simulates a virtualelement (e.g., flight, land and/or marine vehicle, healthcare-relatedelement, etc.) comprising a plurality of dynamic subsystems. Thesimulation mapping system comprises a processor module (e.g., 1130,1830) that obtains dynamic data related to the virtual element beingsimulated in an interactive computer simulation station comprising atangible instrument module. The dynamic data captures actions performedby the user during the training activity on one or more tangibleinstruments of the tangible instrument module (1160). The processormodule also constructs a dataset corresponding to the plurality ofperformance metric values from the dynamic data having a target timestep by synchronizing dynamic data from at least two of the dynamicsubsystems into the dataset considering the target time step, the atleast two of the dynamic subsystems being associated to at least onecommon performance metric values from the plurality of performancemetric values and by inferring, for at least one missing dynamicsubsystems of the plurality of dynamic subsystems missing from thedynamic data, a new set of data into the dataset from dynamic dataassociated to one or more co-related dynamic subsystems, the co-relateddynamic subsystems and the at least one missing dynamic subsystemsimpacting at least one common performance metric values from theplurality of performance metric values.

The processor module may optionally obtain dynamic data from a pluralityof interactive computer simulation stations and constructs the datasethaving the target time step for the plurality of interactive computersimulation stations.

The processor module may optionally further provide the dataset as acommon standardized stream consumers, the consumers comprising a gradingsystem. The common standardized stream may comprise classificationinformation related to the plurality of performance metric values.

The processor module may optionally, when constructing the datasetcorresponding to the plurality of performance metric values from thedynamic data having the target time, add at least one simulated dynamicsubsystem missing from the dynamic data and an additional set of datainto the dataset from dynamic data associated to one or more co-relateddynamic subsystems, the co-related dynamic subsystems and the at leastone simulated dynamic subsystems impacting the at least one commonperformance metric values from the plurality of performance metricvalues.

The processor module may optionally apply a linear quadratic estimation(LQE) when constructing the dataset and/or a probabilistic directedacyclic graphical model when constructing the dataset.

In the first set of embodiments a second aspect is directed to a method2000 for determining a plurality of performance metric values inrelation to a training activity performed by a user in an interactivecomputer simulation, the interactive computer simulation simulating avirtual element comprising a plurality of dynamic subsystems. The method2000 comprises obtaining 2020 dynamic data related to the virtualelement being simulated in an interactive computer simulation stationcomprising a tangible instrument module. The dynamic data capturesactions performed by the user during the training activity on one ormore tangible instruments of the tangible instrument module provided2010 to the user. The method 2000 also comprises constructing a datasetcorresponding to the plurality of performance metric values from thedynamic data having a target time step by synchronizing 2030 dynamicdata from at least two of the dynamic subsystems into the datasetconsidering the target time step, the at least two of the dynamicsubsystems being associated to at least one common performance metricvalues from the plurality of performance metric values and inferring2040, for at least one missing dynamic subsystem of the plurality ofdynamic subsystems missing from the dynamic data, a new set of data intothe dataset from dynamic data associated to one or more co-relateddynamic subsystems, the co-related dynamic subsystems and the at leastone missing dynamic subsystems impacting the at least one commonperformance metric values from the plurality of performance metricvalues. 2020, 2030 and 2040 are repeated (2050) as needed consideringthe behavior of the virtual element/the user in the interactive computersimulation.

The method 2000 may optionally further comprise obtaining dynamic datafrom a plurality of interactive computer simulation stations, whereinconstructing the dataset having the target time step is performed forthe plurality of interactive computer simulation stations.

The method 2000 may optionally further comprise providing the dataset asa common standardized stream consumers, the consumers comprising agrading system. The common standardized stream may optionally compriseclassification information related to the plurality of performancemetric values.

The method 2000 may optionally further comprise, when constructing thedataset corresponding to the plurality of performance metric values fromthe dynamic data having the target time, adding at least one simulateddynamic subsystem missing from the dynamic data and an additional set ofdata into the dataset from dynamic data associated to one or moreco-related dynamic subsystems, the co-related dynamic subsystems and theat least one simulated dynamic subsystems impacting the at least onecommon performance metric values from the plurality of performancemetric values.

Optionally, constructing the dataset may performed by applying a linearquadratic estimation (LQE) and/or by applying a probabilistic directedacyclic graphical model when constructing the dataset.

Reference is now made to FIGS. 1, 3 and 5 to 8 depicting a second set ofembodiments. In the depicted example, an interactive computer simulationsystem (e.g., 1000) is provided for training a user in an interactivecomputer simulation in the performance of a task through a trainingactivity, the interactive computer simulation simulating a virtualelement. The interactive computer simulation system comprises aninteractive computer simulation station (e.g., 1100) and a processormodule (e.g., 1130, 1830). The interactive computer simulation stationcomprises a tangible instrument module (e.g., 1160), the userinteracting with the tangible instrument module for controlling thevirtual element in the interactive computer simulation.

The processor module obtains a plurality of performance metric datasetsrelated to the virtual element being simulated, the plurality ofperformance metric datasets representing results of the interactionsbetween the user and the tangible instrument module and, duringexecution of the interactive computer simulation, detects, in theplurality of performance metric datasets, a plurality of actualmaneuvers of the virtual element during the training activity,identifies one or more standard operating procedures (SOP) for thetraining activity from a plurality of the individually detected actualmaneuvers, provides, in real-time upon detection of the SOPs,information for display in the interactive computer simulation relatedthe SOPs.

The system may optionally further comprise a simulation mapping system(e.g., 1800) for determining a plurality of performance metric values inrelation to the training activity performed by the user in theinteractive computer simulation, the interactive computer simulationsimulating the virtual element comprising a plurality of dynamicsubsystems. The plurality of performance metric datasets may be providedby the simulation mapping system.

The processor module may further obtain a scorecard related to thetraining activity to establish a list of the one or more SOPs ofinterest. The one or more SOPs may identify the plurality of theindividually detected actual maneuvers related thereto.

The plurality of performance metric datasets related to the virtualelement being simulated may be used to provide a grading scorecard forthe detected SOPs. The information for display in the interactivecomputer simulation related the SOPs may further comprise the gradingscorecard for the detected SOPs. The detected actual maneuvers may belogged for post-activity debriefing.

In an optional embodiment, the processor module further obtains aplurality of expected maneuvers of the virtual element during thetraining activity, the plurality of expected maneuvers comprising aplurality of expected individual maneuvers expected and one or morenested maneuvers formed by more than one individual maneuvers from theplurality of expected individual maneuvers, computes the plurality ofperformance metric datasets to identify actual maneuvers of the virtualelement during the training activity, identifies and grades one or moreactual nested maneuvers against corresponding ones of the expectednested maneuvers and, notwithstanding performance of the actual nestedmaneuvers, identifies and grades a plurality of actual individualmaneuvers against the plurality of expected individual maneuvers.

In the second set of embodiments, a second aspect is directed to aninteractive computer simulation station (e.g., 1100) for training a userin an interactive computer simulation in the performance of a taskthrough a training activity, the interactive computer simulationsimulating a virtual element. The interactive computer simulationstation comprises a tangible instrument module 1160, the userinteracting with the tangible instrument module for controlling thevirtual element in the interactive computer simulation and a processormodule.

The processor module (e.g., 1130) obtains a plurality of performancemetric datasets related to the virtual element being simulated, theplurality of performance metric datasets representing results of theinteractions between the user and the tangible instrument module and,during execution of the interactive computer simulation, detects, in theplurality of performance metric datasets, a plurality of actualmaneuvers of the virtual element during the training activity,identifies one or more standard operating procedures (SOP) for thetraining activity from a plurality of the individually detected actualmaneuvers and provides, in real-time upon detection of the SOPs,information for display in the interactive computer simulation relatedthe SOPs.

The interactive computer simulation station may further comprise anetwork interface nodule (e.g., 1140) for receiving the plurality ofperformance metric datasets from a simulation mapping system thatdetermines a plurality of performance metric values in relation to thetraining activity performed by the user in the interactive computersimulation.

The processor module may optionally further obtain a scorecard relatedto the training activity to establish a list of the one or more SOPs ofinterest. The one or more SOPs may further identify the plurality of theindividually detected actual maneuvers related thereto.

The plurality of performance metric datasets related to the virtualelement being simulated may be used to provide a grading scorecard forthe detected SOPs. The information for display in the interactivecomputer simulation related the SOPs may comprise the grading scorecardfor the detected SOPs.

In an optional embodiment, the processor module further obtains aplurality of expected maneuvers of the virtual element during thetraining activity, the plurality of expected maneuvers comprising aplurality of expected individual maneuvers expected and one or morenested maneuvers formed by more than one individual maneuvers from theplurality of expected individual maneuvers, computes the plurality ofperformance metric datasets to identify actual maneuvers of the virtualelement during the training activity, identifies and grades one or moreactual nested maneuvers against corresponding ones of the expectednested maneuvers and, notwithstanding performance of the actual nestedmaneuvers, identifies and grades a plurality of actual individualmaneuvers against the plurality of expected individual maneuvers.

In the second set of embodiments a third aspect is directed to a method3000 for training a user in an interactive computer simulation in theperformance of a task through a training activity, the interactivecomputer simulation simulating a virtual element. The method 3000comprises in an interactive computer simulation station, providing 3010a tangible instrument module (e.g., 1160) to the user for controllingthe virtual element in the interactive computer simulation. The method3000 also comprises obtaining 3020 a plurality of performance metricdatasets related to the virtual element being simulated, the pluralityof performance metric datasets representing results of the interactionsbetween the user and the tangible instrument module and, duringexecution of the interactive computer simulation at the interactivecomputer simulation station, detecting 3030, in the plurality ofperformance metric datasets, one or more actual maneuvers of the virtualelement during the training activity, identifying 3040 one or morestandard operating procedures (SOP) from the detected actual maneuversand displaying 3050, in real-time upon detection of the SOPs,information in the interactive computer simulation related the SOPs.3020, 3030, 3040 and optionally 3060 may be repeated 3060 multipletimes, depending on the behavior of the virtual element/the user in theinteractive computer simulation.

The method 3000 may further optionally comprise determining, at asimulation mapping system, a plurality of performance metric values inrelation to the training activity performed by the user in theinteractive computer simulation, the interactive computer simulationsimulating the virtual element comprising a plurality of dynamicsubsystems. The plurality of performance metric datasets may be providedby the simulation mapping system.

The method 3000 may further optionally comprise obtaining a scorecardrelated to the training activity to establish a list of the one or moreSOPs of interest. The one or more SOPs may further identify theplurality of the individually detected actual maneuvers related thereto.

The plurality of performance metric datasets related to the virtualelement being simulated may be used to provide a grading scorecard forthe detected SOPs. The information for display in the interactivecomputer simulation related the SOPs may then optionally comprise thegrading scorecard for the detected SOPs.

The method 3000 may further optionally comprise logging the detectedactual maneuvers and debriefing the training activity from the loggeddetected actual maneuvers.

In some embodiments, the method 3000 may further optionally compriseobtaining a plurality of expected maneuvers of the virtual elementduring the training activity, the plurality of expected maneuverscomprising a plurality of expected individual maneuvers expected and oneor more nested maneuvers formed by more than one individual maneuversfrom the plurality of expected individual maneuvers, computing theplurality of performance metric datasets to identify actual maneuvers ofthe virtual element during the training activity, identifying and gradesone or more actual nested maneuvers against corresponding ones of theexpected nested maneuvers and, notwithstanding performance of the actualnested maneuvers, identifying and grading a plurality of actualindividual maneuvers against the plurality of expected individualmaneuvers.

Reference is now made to FIGS. 1 and 4 to 8 depicting a third set ofembodiments. An interactive computer-based training system (e.g., 1000)is depicted for assessing a training activity performed by a user in aninteractive computer simulation, the interactive computer simulationsimulating a virtual element. The training system comprises aninteractive computer simulation station (e.g., 1100) comprising atangible instrument module (e.g., 1160), the user interacting with thetangible instrument module for controlling the virtual element in theinteractive computer simulation and a processor module (e.g., 1130,1830). The processor module obtains a plurality of performance metricdatasets related to the virtual element being simulated the interactivecomputer simulation station, the plurality of performance metricdatasets representing results of the interactions between the user andthe tangible instrument module, obtains a plurality of expectedmaneuvers of the virtual element during the training activity, theplurality of expected maneuvers, computes the plurality of performancemetric datasets to identify actual maneuvers of the virtual elementduring the training activity, identifies one or more failed actualmaneuvers of the virtual element during the training activity againstcorresponding ones of the expected maneuvers and performs computationalregression on the actual maneuvers of the virtual element compared tothe expected maneuvers of the virtual element to identify one or moreroot causes of the failed actual maneuvers, the computational regressionbeing performed on the actual maneuvers notwithstanding thecorresponding expected maneuvers being met thereby.

The system may further optionally comprise a simulation mapping systemfor determining a plurality of performance metric values in relation tothe training activity performed by the user in the interactive computersimulation, the interactive computer simulation simulating the virtualelement comprising a plurality of dynamic subsystems, wherein theplurality of performance metric datasets is provided by the simulationmapping system.

The processor module may further optionally map, in real-time, each oneof the actual maneuvers of the virtual element during the trainingactivity on causal model for linking the one actual maneuver withprevious ones of the actual maneuvers. The processor module may thenoptionally associate a probability rating to the one or more root causesof the failed actual maneuvers considering the causal model.

The processor module may further optionally provide to an instructor ofthe user, in real-time, the one or more root causes of the failed actualmaneuvers.

The plurality of performance metric datasets related to the virtualelement being simulated may be used to provide a grading scorecard forthe actual maneuvers. The grading scorecard for the actual maneuvers maybe provided for display in the interactive computer simulation.

In the third set of embodiments a second aspect is directed to aninteractive computer simulation station (e.g., 1100) for assessing atraining activity performed by a user in an interactive computersimulation, the interactive computer simulation simulating a virtualelement. The interactive computer simulation station comprises atangible instrument module (e.g., 1160), the user interacting with thetangible instrument module for controlling the virtual element in theinteractive computer simulation and a processor module. The processormodule (e.g., 1130) obtains a plurality of performance metric datasetsrelated to the virtual element being simulated the interactive computersimulation station, the plurality of performance metric datasetsrepresenting results of the interactions between the user and thetangible instrument module, obtains a plurality of expected maneuvers ofthe virtual element during the training activity, the plurality ofexpected maneuvers, computes the plurality of performance metricdatasets to identify actual maneuvers of the virtual element during thetraining activity, identifies one or more failed actual maneuvers of thevirtual element during the training activity against corresponding onesof the expected maneuvers and performs computational regression on theactual maneuvers of the virtual element compared to the expectedmaneuvers of the virtual element to identify one or more root causes ofthe failed actual maneuvers, the computational regression beingperformed on the actual maneuvers notwithstanding the correspondingexpected maneuvers being met thereby.

The interactive computer simulation station may further comprise anetwork interface nodule (e.g., 1140) for receiving the plurality ofperformance metric datasets from a simulation mapping system thatdetermines a plurality of performance metric values in relation to thetraining activity performed by the user in the interactive computersimulation.

The processor module may further optionally map, in real-time, each oneof the actual maneuvers of the virtual element during the trainingactivity on causal model for linking the one actual maneuver withprevious ones of the actual maneuvers and associate a probability ratingto the one or more root causes of the failed actual maneuversconsidering the causal model.

The processor module may further optionally provide to an instructor ofthe user, in real-time, the one or more root causes of the failed actualmaneuvers.

The plurality of performance metric datasets related to the virtualelement being simulated may be used to provide a grading scorecard forthe actual maneuvers. The grading scorecard for the actual maneuvers maythen be provided for display in the interactive computer simulation.

In the third set of embodiments a third aspect is directed to a method4000 is depicted for assessing a training activity performed by a userin an interactive computer simulation, the interactive computersimulation simulating a virtual element. The method 4000 comprisesobtaining 4020 a plurality of performance metric datasets related to thevirtual element being simulated the interactive computer simulationstation, the plurality of performance metric datasets representingresults of interactions of the user provided 4010 with a tangibleinstrument module (e.g., 1160) in an interactive computer simulationstation, the user interacting with the tangible instrument module forcontrolling the virtual element in the interactive computer simulation,obtaining 4030 a plurality of expected maneuvers of the virtual elementduring the training activity, computing 4040 the plurality ofperformance metric datasets to identify actual maneuvers of the virtualelement during the training activity, identifying 4050 one or morefailed actual maneuvers of the virtual element during the trainingactivity against corresponding ones of the expected maneuvers andperforming 4060 computational regression on the actual maneuvers of thevirtual element compared to the expected maneuvers of the virtualelement to identify one or more root causes of the failed actualmaneuvers, the computational regression being performed on the actualmaneuvers notwithstanding the corresponding expected maneuvers being metthereby.

The method 4000 may optionally further comprise determining, at asimulation mapping system (e.g., 1800), a plurality of performancemetric values in relation to the training activity performed by the userin the interactive computer simulation, the interactive computersimulation simulating the virtual element comprising a plurality ofdynamic subsystems, wherein the plurality of performance metric datasetsis provided by the simulation mapping system.

The method 4000 may optionally further comprise mapping, in real-time,each one of the actual maneuvers of the virtual element during thetraining activity on causal model for linking the one actual maneuverwith previous ones of the actual maneuvers. The method 4000 may thenfurther comprise associating a probability rating to the one or moreroot causes of the failed actual maneuvers considering the causal modeland providing to an instructor of the user, in real-time, the one ormore root causes of the failed actual maneuvers. The plurality ofperformance metric datasets related to the virtual element beingsimulated may be used to provide a grading scorecard for the actualmaneuvers. The method 4000 may optionally further comprise providing4070 for display in the interactive computer simulation the gradingscorecard for the actual maneuvers. 4020, 4030, 4040, 4050, 4060 andoptionally 4070 may be repeated 4080 multiple times, depending on thebehavior of the virtual element/the user in the interactive computersimulation.

A method is generally conceived to be a self-consistent sequence ofsteps leading to a desired result. These steps require physicalmanipulations of physical quantities. Usually, though not necessarily,these quantities take the form of electrical or magnetic/electromagneticsignals capable of being stored, transferred, combined, compared, andotherwise manipulated. It is convenient at times, principally forreasons of common usage, to refer to these signals as bits, values,parameters, items, elements, objects, symbols, characters, terms,numbers, or the like. It should be noted, however, that all of theseterms and similar terms are to be associated with the appropriatephysical quantities and are merely convenient labels applied to thesequantities. The description of the present invention has been presentedfor purposes of illustration but is not intended to be exhaustive orlimited to the disclosed embodiments. Many modifications and variationswill be apparent to those of ordinary skill in the art. The embodimentswere chosen to explain the principles of the invention and its practicalapplications and to enable others of ordinary skill in the art tounderstand the invention in order to implement various embodiments withvarious modifications as might be suited to other contemplated uses.

What is claimed is:
 1. A simulation mapping system for determining aplurality of performance metric values in relation to a trainingactivity performed by a user in an interactive computer simulation, theinteractive computer simulation simulating a virtual element comprisinga plurality of dynamic subsystems, the simulation mapping systemcomprising: a processor module configured to: obtain dynamic datarelated to the virtual element being simulated in an interactivecomputer simulation station comprising a tangible instrument module,wherein the dynamic data captures actions performed by the user duringthe training activity on one or more tangible instruments of thetangible instrument module; and construct a dataset corresponding to theplurality of performance metric values from the dynamic data having atarget time step by: synchronizing dynamic data from at least two of thedynamic subsystems into the dataset considering the target time step,the at least two of the dynamic subsystems being associated to at leastone common performance metric values from the plurality of performancemetric values; and inferring, for at least one missing dynamicsubsystems of the plurality of dynamic subsystems missing from thedynamic data, a new set of data into the dataset from dynamic dataassociated to one or more co-related dynamic subsystems, the co-relateddynamic subsystems and the at least one missing dynamic subsystemsimpacting at least one common performance metric values from theplurality of performance metric values.
 2. The simulation mapping systemof claim 1, wherein the processor module is configured to obtain dynamicdata from a plurality of interactive computer simulation stations andconstruct the dataset having the target time step for the plurality ofinteractive computer simulation stations.
 3. The simulation mappingsystem of claim 1, wherein processor module is further configured toprovide the dataset as a common standardized stream consumers, theconsumers comprising a grading system.
 4. The simulation mapping systemof claim 3, wherein the common standardized stream comprisesclassification information related to the plurality of performancemetric values.
 5. The simulation mapping system of claim 1, wherein theprocessor module is configured to, when constructing the datasetcorresponding to the plurality of performance metric values from thedynamic data having the target time, add at least one simulated dynamicsubsystem missing from the dynamic data and an additional set of datainto the dataset from dynamic data associated to one or more co-relateddynamic subsystems, the co-related dynamic subsystems and the at leastone simulated dynamic subsystems impacting the at least one commonperformance metric values from the plurality of performance metricvalues.
 6. The simulation mapping system of claim 1, wherein theprocessor module is configured to apply a linear quadratic estimation(LQE) when constructing the dataset.
 7. The simulation mapping system ofclaim 1, wherein the processor module is configured to apply aprobabilistic directed acyclic graphical model when constructing thedataset.
 8. A method for determining a plurality of performance metricvalues in relation to a training activity performed by a user in aninteractive computer simulation, the interactive computer simulationsimulating a virtual element comprising a plurality of dynamicsubsystems, the method comprising: obtaining dynamic data related to thevirtual element being simulated in an interactive computer simulationstation comprising a tangible instrument module, wherein the dynamicdata captures actions performed by the user during the training activityon one or more tangible instruments of the tangible instrument module;and constructing a dataset corresponding to the plurality of performancemetric values from the dynamic data having a target time step by:synchronizing dynamic data from at least two of the dynamic subsystemsinto the dataset considering the target time step, the at least two ofthe dynamic subsystems being associated to at least one commonperformance metric values from the plurality of performance metricvalues; and inferring, for at least one missing dynamic subsystem of theplurality of dynamic subsystems missing from the dynamic data, a new setof data into the dataset from dynamic data associated to one or moreco-related dynamic subsystems, the co-related dynamic subsystems and theat least one missing dynamic subsystems impacting the at least onecommon performance metric values from the plurality of performancemetric values.
 9. The method of claim 8, further comprising obtainingdynamic data from a plurality of interactive computer simulationstations, wherein constructing the dataset having the target time stepis performed for the plurality of interactive computer simulationstations.
 10. The method of claim 8, further comprising providing thedataset as a common standardized stream consumers, the consumerscomprising a grading system.
 11. The method of claim 10, wherein thecommon standardized stream comprises classification information relatedto the plurality of performance metric values.
 12. The method of claim8, further comprising, when constructing the dataset corresponding tothe plurality of performance metric values from the dynamic data havingthe target time, adding at least one simulated dynamic subsystem missingfrom the dynamic data and an additional set of data into the datasetfrom dynamic data associated to one or more co-related dynamicsubsystems, the co-related dynamic subsystems and the at least onesimulated dynamic subsystems impacting the at least one commonperformance metric values from the plurality of performance metricvalues.
 13. The method of claim 8, wherein constructing the dataset isperformed by applying a linear quadratic estimation (LQE).
 14. Themethod of claim 8, wherein constructing the dataset is performed byapplying a probabilistic directed acyclic graphical model whenconstructing the dataset.
 15. A non-transitory computer-readable mediumhaving computer-readable instructions stored thereon, which, whenexecuted by a processor of a computer system, configures the computersystem to perform a method for determining a plurality of performancemetric values in relation to a training activity performed by a user inan interactive computer simulation, the interactive computer simulationsimulating a virtual element comprising a plurality of dynamicsubsystems, the method comprising: obtaining dynamic data related to thevirtual element being simulated in an interactive computer simulationstation comprising a tangible instrument module, wherein the dynamicdata captures actions performed by the user during the training activityon one or more tangible instruments of the tangible instrument module;and constructing a dataset corresponding to the plurality of performancemetric values from the dynamic data having a target time step by:synchronizing dynamic data from at least two of the dynamic subsystemsinto the dataset considering the target time step, the at least two ofthe dynamic subsystems being associated to at least one commonperformance metric values from the plurality of performance metricvalues; and inferring, for at least one missing dynamic subsystem of theplurality of dynamic subsystems missing from the dynamic data, a new setof data into the dataset from dynamic data associated to one or moreco-related dynamic subsystems, the co-related dynamic subsystems and theat least one missing dynamic subsystems impacting the at least onecommon performance metric values from the plurality of performancemetric values.
 16. The non-transitory computer-readable medium of claim15 wherein the method further comprises obtaining dynamic data from aplurality of interactive computer simulation stations, and whereinconstructing the dataset having the target time step is performed forthe plurality of interactive computer simulation stations.
 17. Thenon-transitory computer-readable medium of claim 15 wherein the methodfurther comprises providing the dataset as a common standardized streamconsumers, the consumers comprising a grading system.
 18. Thenon-transitory computer-readable medium of claim 17, wherein the commonstandardized stream comprises classification information related to theplurality of performance metric values.
 19. The non-transitorycomputer-readable medium of claim 15 wherein the method furthercomprises when constructing the dataset corresponding to the pluralityof performance metric values from the dynamic data having the targettime, adding at least one simulated dynamic subsystem missing from thedynamic data and an additional set of data into the dataset from dynamicdata associated to one or more co-related dynamic subsystems, theco-related dynamic subsystems and the at least one simulated dynamicsubsystems impacting the at least one common performance metric valuesfrom the plurality of performance metric values.
 20. The non-transitorycomputer-readable medium of claim 15 wherein constructing the dataset isperformed by applying a linear quadratic estimation (LQE).